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A  model  is  presented  for  situation  and  threat  assessment,  with  a  goal  of  advancing  the  state  of  the  art  in 
representing,  recognizing  and  discovering  situations  and,  in  particular,  threat  situations. 

The  activity  relates  to  levels  2  and  3  of  the  familiar  JDL  data  fusion  model.  Level  2,  “Situation  Assessment”, 
involves  such  applications  as  scene  understanding,  force  structure  analysis  and  many  other  types  of  situational 
analysis.  Level  3,  “Impact  Assessment”  includes,  besides  threat  assessment  (as  level  3  was  originally  named), 
course  of  action  analysis  and  outcome  prediction. 

Data  fusion  is  the  process  of  estimating  or  predicting  some  aspect  of  the  world.  Specifically,  the  data  fusion 
process  of  Situation  Assessment  has  the  job  of  estimating  or  predicting  situations.  A  situation  can  be  defined 
very  broadly  as  “any  structure  part  of  reality”  (Devlin,  1991).  In  that  structural  analysis  involves  an 
assessment  of  the  element  of  an  entity  in  their  relation  to  one  another,  Situation  Assessment  involves  (a) 
inferring  relationships,  (b)  inferring  the  states  of  elements  on  the  basis  of  estimates  of  their  relationships,  and 
(c)  recognizing  or  classifying  situations  on  the  basis  of  estimates  of  constituent  elements  and  their 
relationships.  In  that  the  last  of  these  is  a  recognition/classification  problem,  we  should  expect  to  have  some 
similarity  to  target  recognition  and  classification;  i.e.  the  matching  of  data  to  prior  models  (a  deductive 
process).  As  in  target  recognition/classification,  this  dependency  on  prior  models  presumes  a  process  for 
generating,  evaluating  and  selecting  such  models.  These  are  characteristically  abductive  (i.e.  explanatory)  and 
inductive  processes. 

We  take  as  our  starting  point  the  process  involved  in  Scene  Understanding  as  it  occurs  in  machine  vision, 
automatic  target  recognition  (ATR)  and  remote  sensing  applications.  ATR  and  machine  vision  have  evolved 
from  straight-forward  template  matching  techniques,  in  which  observed  scenes  are  compared  with  stored 
images.  Template  techniques  are  obviously  constrained  by  the  number  of  target/context  scenes  that  they  can 
store.  Various  indexing  schemes  are  used  to  reduce  this  burden,  by  extracting  relatively  invariant  features,  but 
the  approach  is  ultimately  restricted  to  situations  in  which  target  signatures  are  not  much  affected  by 
contextual  factors  (occlusion,  shadowing,  illumination  variability,  etc.). 

Far  more  robust  (although  more  complex)  are  model-based  techniques,  in  which  candidate  scene  hypotheses 
are  adaptively  generated,  evaluated,  refined  and  modified.  Such  techniques  can  be  augmented  by  adaptive  data 
collection  techniques  that  anticipate  the  utility  of  information  in  resolving  present  ambiguities  and  manage 
sensors  or  data  mining  processes  to  maximize  the  value  of  returned  data. 
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To  extend  model-based  scene  understanding  processes  for  generalized  Situation  Assessment  problem,  the 
technique  needs  to  be  able  to  “think  outside  the  frame”,  as  evidence  can  come  from  “anywhere”.  Furthermore, 
the  process  must  often  deal  with  a  wider  range  of  evidence:  understanding  of  many  types  of  situations 
involves  the  use  of  a  wide  diversity  of  information  types,  relating,  e.g.  to  political,  social,  informational  and 
psychological  aspects  of  the  situation  that  are  not  inferable  from  direct  physical  signatures  alone. 
Correspondingly,  the  recognition/classification  process  must  often  involve  a  wider  range  of  entity  and 
aggregate  behavior  models.  This  is  certainly  the  case  in  the  many  situations  of  interest  that  involve  estimating 
and  predicting  human  individual  and  group  behavior. 

Therefore  it  will  be  necessary  to  have  deductive,  abductive  and  inductive  tools  that  will  characterize  and 
recognize  situations  and  constituent  relationships.  In  various  applications,  relationships  of  interest  can  range 
from  logical  and  semantic  ones,  to  physical,  functional,  conventional  and  cognitive  relationships.  To  be  sure, 
most  relationships  of  these  types  are  not  Directly  Observable;  rather,  they  must  be  inferred  from  the 
characteristics  &  activities  of  the  related  entities. 

We  are  working  on  defining  a  formal  scheme  for  representing  and  reasoning  about  relationships,  situations, 
impacts,  threat,  etc.  First,  it  should  be  noted  that  a  relationships  is  not  simply  a  multi-target  state:  a  state  of  the 
sort  of  interest  in  Situation  Assessment  cannot  in  general  be  inferred  from  a  set  of  single-target  states.  We  will 
want  to  combine  both  direct  measurements  from  “sensors”  as  all  as  inferred  information  -  e.g.  natural 
language  reports  by  an  analyst,  or  from  HUMINT,  document  or  communications.  All  such  information  can  be 
represented  in  a  second-order  predicate  calculus.  We  use  the  formalism  of  “Infons”  introduced  by  Keith 
Devlin  (1991),  but  there  are  equivalent  formalisms.  Situations  s  can  be  defined  by  means  of  minimal  sets  ji(s) 
of  infons,  often  with  fuzzy  membership  conditions.  Infons  and  situations  can  be  elements  of  other  infons.  This 
formulation  supports  Operational  Net  Assessment:  reasoning  from  multiple  perspectives. 

Elements  of  state  estimation  relevant  to  threat  assessment  (and,  indeed,  to  many  other  types  of  situation  and 
impact  assessment)  extend  beyond  the  physical  state  elements  familiar  in  target  tracking  and  target 
recognition.  Ed  Waltz,  following  Karl  Popper,  has  identified  physical,  informational  and  perceptual  aspects  of 
state  estimation  problems.  We  have  used  this  to  develop  a  taxonomy  of  discrete  and  continuous  state  elements 
for  physical,  informational  and  perceptual  aspects,  both  of  entities  and  of  their  relationships.  Reasoning  about 
informational  and  perceptual  (and,  more  broadly,  psychological)  aspects  are  at  the  heart  of  Operational  Net 
Assessment. 

We  turn  to  the  estimation  and  prediction  of  threats',  or,  more  generally,  of  intentional  acts.  Little  and  Rogova, 
following  the  tradition  of  forensic  analysis,  have  defined  threats  (i.e.  the  potential  for  harmful  actions)  as  a 
product  of  the  capability,  opportunity  and  intent  of  agents  to  carry  out  such  actions  (We  can  generalize  this  to 
encompass  any  intentional  act;  we  can  further  generalize  this  to  include  non-intentional  events  -  harmful  or 
otherwise  -  by  ignoring  intent). 

The  process  defined  for  model-based  scene  understanding  is  extended  to  the  general  problem  of  situation  and 
threat  assessment.  This  is  done  in  a  very  straight-forward  manner,  by  expanding  the  types  of  modes  for 
information  sources,  targets  and  situations.  The  functions  of  model  management  are  also  explicitly  addressed. 
A  consistent  scheme  for  representing  measurements,  attributive  and  relational  states  and  corresponding 
uncertainties  is  required  throughout  the  deductive  recognition/classification  processes  and  the 
abductive/inductive  model  management  processes. 

The  problem  of  characterizing  information  sources  is  present  even  in  systems  that  have  been  designed  and 
integrated  as  a  package.  Sensor  noise  statistics  and  biases  can  be  characterized  in  calibration  and  registration; 
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either  by  using  (a)  internal  calibration  signal  sources,  (b)  known  “fiducial”  targets  in  the  environment,  or  (c) 
the  ensemble  of  information  available  in  a  multi-sensor  system. 

The  problem  is  made  more  difficult  when  the  performance  of  information  sources  cannot  be  assumed.  This  is 
the  case  in  network-centric  operations,  in  which  calibration  and  registration  are  not  easily  performed.  It  is 
even  more  the  case  when  the  cooperation  of  information  sources  cannot  be  assured;  e.g.  when  sources  are 
agents  with  varying  degrees  of  autonomy,  raising  the  possibility  of  private  agendas.  The  same  factors  can 
affect  the  estimation  and  controllability  of  assets  having  some  degree  of  autonomy.  Cases  extend  to  the  use  of 
non-cooperating  agents;  e.g.  enemy  radars  that  “report”  to  our  ELINT  systems  who  and  where  they  are; 
generally  with  no  attempt  to  deceive.  Other  cases  include  deception,  in  such  Information  Warfare  techniques 
as  decoys,  deceptive  jamming  and  propaganda.  Then  there  are  third-party  agents  -  e.g.  commercial  news 
sources,  reference  texts,  and  the  like  -  that  may  or  may  not  be  unbiased. 

Given  the  general  lack  of  useful  calibration  sources  or  fiducial  targets  in  such  cases,  fusion  system  must 
characterize  the  reliability  &  performance  of  EACH  source  using  the  ensemble  of  information.  In  other  words, 
the  sources  available  to  information  systems  (and  control  systems,  too)  can  involve  various  degrees  of  noise 
and  bias  errors  AND  various  degrees  of  performance  characterization.  To  extend  the  network-centric  concept, 
this  implies  that  we  can’t  assume  clear  system  boundaries:  “us”  vs  “them”:  Allegiance  is  a  matter  of 
estimation. 

Technical  issues  in  situation  and  threat  assessment  include:  (a)  Data  Alignment:  Exploiting  heterogeneous 
information  types  and  Normalizing  confidence:  source  characterization,  data  evaluation;  (b)  Hypothesis 
Generation:  determining  relevant  data,  generating/selecting  candidate  scenario  hypotheses  and  representing 
relationships  ,  situations,  etc.;  (c)  Hypothesis  Evaluation:  managing  uncertainties  in  data  association;  (d) 
Hypothesis  Selection:  efficient  schemes  for  representing  and  searching  hypotheses. 

Engineering  issues  include  principles  for  partitioning  the  process  into  an  information  exploitation  architecture. 
This  includes  issues  of  allocating  tasks  to  humans  and  to  automated  processes  and  attendant  presentation  and 
control  issues.  Particularly  difficult  engineering  issues  involve  managing  models  of  situations,  threats,  sources 
and  system  resources.  Such  model  management  and  exploitation  will  require  an  otology  of  situational, 
mission  and  system  elements  and  their  relationships.  The  ontology  will  need  to  provide  criteria  for 
recognizing  relationships  and  situations.  It  must  capture  uncertainties  in  dependencies  (e.g.  in  terms  of  fuzzy 
membership). 
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Objectives 


*  Advance  the  state-of-the  art  in 

-  Representing 

-  Recognizing 

-  Discovering 

situations  of  interest;  particularly  threat  situations 

*  Improve  inferencing*  across  information  levels: 

Level  0.  Measurements  &  Features 

Level  1.  Entities 

Level  2.  Relationships  &  Situations 

Level  3.  Outcomes  &  Costs 

Level  4.  Own  System  Characteristics 


*  By  people,  machine  or  some  combination 


Characteristic  Process  Flow 
Across  the  Data  Fusion  “Levels” 


•  Collection  Management 

•  Data  Mining 

•  Fusion  Process  Control 

•  Threat  Assessment 

•  Course  of  Action  Analysis 

•  Outcome  Prediction 

Force  Structure  Analysis 
Situation  Analysis 
Scene  Understanding 

Target  Recognition 
Target  Tracking 


•  Feature  Extraction 

•  Signal  Definition 


Plans 

1  Goals 

Level  4 

Sys  Characterization/Process  Refinement 


Impact 

Plans,  Sub-Goals 

/  ■  Level  3 

Impact  (Threat)  Assessmer 

/ 

Situations 


Utility 


Level  2 

Situation  Assessment 


Objects 

Situations 

Level  1  ' 

Object  Assessment 

W 

.1 

t  Signals/Features 

Objects 

Signals/Fea  tures 
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Definition  of  Terms 


•  Data  Fusion  :=  estimation/prediction  of  the  state  of  some 
aspect  of  the  world  on  the  basis  of  multiple  data  ( JDL  model, 
1998  revision) 

•  Situation  :=  any  structured  part  of  reality  ( K .  Devlin,  1991) 

•  Structure  :=  the  aggregate  of  elements  of  an  entity  in  their 
relationships  to  each  other  (Merriam-Webster  Online 
Dictionary,  2005) 

•  Situation  Assessment  (Level  2  Data  Fusion)  := 
estimation/prediction  of  the  state  of  some  situation 

a.  Inferring  relationships 

b.  Inferring  the  states  of  elements  on  the  basis  of  estimates  of 
their  relationships 

c.  Recognizing/classifying  situations  on  the  basis  of  estimates 
of  constituent  elements  and  their  relationships 


Level  2/3  Inference  Patterns 


1 .  Inferring  relationships  on  the  basis  of  entity  states 

Ri(x),  R2(y)  R3(x,y), 


2.  Inferring  the  presence  and  the  states  of  entities  on  the  basis  of 
relations  in  which  they  participate 

SAM  Battery 


Fire  Control 
Radar 


3.  Inferring  relationships  on  the  basis  of  other  relationships 
Communicating  (x,y)  ->  Cooperating  (x,y) 

4.  Recognizing  and  characterizing  extant  situations 

R4(x,y,z),  R5(z),....  — >  x,y,z  involved  in  situation  S 

5.  Predicting  undetected  (e.g.  future)  situations 

S1  at  timey  — ►  S2  at  time2 

Such  inferences  may  be  conditional,  counterfactual,  or  uncertain 
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Starting  Point:  Model-Based  Evidential 
Accrual  for  Scene  Understanding 


Known  sensing  conditions 


Live  Unknown 
Scene/ 
Situation 


FEATURE 

FEATURE 

FEATURE 

EXTRACTION 

MATCHING 

GENERATION 

Feature 

Requests 


Hyp  Generation 
Hyp  Evaluation 
Hyp  Selection 


Interpretation 

Hypotheses 


Sensor, 
Target  & 
Back¬ 
ground 
Models 


Selected  Scene  Interpretation  Hypothesis 


For  General  Situation  Assessment  Problem, 
need  to 

•  Think  outside  the  frame:  evidence  can 
come  from  “anywhere” 

•  Deal  with  wider  range  of  evidence 

•  (Often)  infer  concerning  human  activity 
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Taxonomy  of  Relationships 
Relevant  to  Situation  Assessment 


•  Logical/Semantic  relationships  (e.g.  definitional,  analytic, 
taxonomic,  mereologic) 

•  Physical  relationships  (e.g.  spatio-temporal,  causal) 

•  Functional  relationships  (e.g.  structural  or  organizational  role) 

•  Conventional  relationships  (e.g.  ownerships,  legal  and  other 
societal  conventions) 

•  Cognitive  relationships  (e.g.  sensing,  perceiving,  believing, 
fearing,  imagining) 


Most  of  these  Relationships  are  not  Directly  Observable: 

Must  be  Inferred  From  the  Characteristics  &  Activities  of 

the  Related  Entities 


Relationships  of  Interest 
in  Tactical  Applications 


Relationships  among  objects  in  threat 
complex  (deployment,  kinetic  interaction, 
organization  role/subordination,  comms,  type 
similarity,  etc.) 

Relationships  among  blue  sensor  &  weapon 
platforms  (spatio-temporal  alignment, 
measurement  calibration,  confidence, 
communication/  coordination,  etc.) 

Relationships  between  sensors  &  sensed 
entities  (intervisibility,  assignment/cueing, 
sensing,  data  association,  countermeasures) 

Relationships  between  red  &  blue  tactical 
entities  (targeting,  jamming,  engaging,  etc.) 


•  Relationships  between  entities  of  interest  & 
other  entities  (terrain  features;  solar  & 
atmospheric  effects;  weapon  launch  &  impact 
points;  etc.) 
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A  Formulation 
for  Situation  Assessment 


•  A  relationships  is  not  simply  a  multi-target  state 

-  A  multi-target  state  of  the  sort  of  interest  in  Situation 
Assessment  cannot  in  general  be  inferred  from  a  set 
of  single-target  states  X=  {x7,...,xj 


E.g.  from  P  =  ‘x  is  healthy,  wealthy  and  wise’ 
can’t  infer  Q  =  ‘x  is  married  to  y’ 
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Formal  Representation  (1) 


•  Want  to  combine  both: 

-  Direct  measurements  -  from  “Sensors” 

-  Inferred  information  -  e.g.  natural  language  reports 
by  an  analyst,  or  from  HUMINT,  documents  or 
communications 

•  All  such  information  can  be  represented  in  a 
second-order  predicate  calculus 

-  H.  Curry  &  R.  Feys,  Combinatory  Logic,  1974 

-  K.  Devlin,  Logic  &  Information,  1991:  “Infon” 
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Formal  Representation  (2) 


*  Infon:  a  unit  of  information  17  =  ( r,x1,...,xn,h,k,p ) 

for  some  m-place  relation  r,  m>n,  entities  Xj,...,xn, 
location  h,  time  k  and  probability  p 

-  Replacing  {0,1}  polarities  with  probabilities  allows  us  to  reason 
with  uncertain  information 

*  Situations  s  can  be  defined  by  means  of  minimal 
sets  fj(s )  of  infons,  often  with  fuzzy  membership 
conditions 

*  Can  classify  real-world  situations  in  the  same  way 
as  we  classify  real-world  objects 

-  Indeed,  an  object  is  -  by  definition  -  a  kind  of  situation 
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Formal  Representation  (2a) 


*  A  Situation-type  or  Object-type  s  can  be 
defined  in  terms  of  an  equivalence  class  R( s) 
of  relationships; 

i.e.  as  a  set  of  un-anchored  infons: 

\v  is  an  s’  can  be  defined  as 

‘V/V'^e  R( s)  ->  3 \y„...,ym_1[(r,x,y1,...,ym_I,h,k,l)]Y 
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Formal  Representation  (3) 


*  Infons  and  situations  can  be  elements  of  other 
infons 

—  E.g.  Sj=  (r,x,  \2, /;,/»,/)).  where  r  =  believes 
(i.e.,  “with  probability  p,  at  place  &  time  h,k,  x believes  that 

V”) 

-  Similarly  with  r  =  perceives,  hypothesizes,  wonders 
whether,  doubts  that,  etc. 

-  Also,  can  be  nested  recursively: 

“x  believes  thaty  believes  that ...  q” 

*  This  formulation  supports  Operational  Net 
Assessment:  reasoning  from  multiple  perspectives 

-  Conditional,  counterfactual,  uncertain 
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Level  2/3  Fusion  Outputs: 
Relational  &  Situational  State  Elements 


•  A  Relationship  Hypothesis  has  the  form 


H  =  (rp  Z1(1),  (rn,  Z1(„)y  ...,Zm(n),Pn),  for  Level  1  hypotheses  Zi9  relation 

types  rjy  and  associated  probabilities pj 

i.e.  “entities  y.  i=1,...,n,  inferred ,  respectively,  from  measurement  sets  Zt  are  in 
relations  Vj  with  probabilities  p” 


•  Level  2  Hypotheses 
(“Perceived 
Relationships”) 


•  Level  1  Hypotheses 
(“Perceived  Entities”) 


•  Measurements 


Discrete  Relational  state 
probability  vector 

<  (*>  Zj(D’  *  *  •’  Zn(j)  ’Pj)’-'  •»> 

Discrete  Attributive  state 
probability  vector 

<  (Type  l,Zn,p1),...,> 

Continuous  state  est 

C K’Zn,S) 


•  Situation  Hypothesis: 

-  A  Set  of  Relationship  Hypotheses  -  often  a  fuzzy  set 


(Infon  notation  with 
suppressed  place  &  time 
indices) 
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Generalized  Belief  Propagation 

—  w  v 


Evidence  accrual  represented  equivalently  via  Belief 
Propagation  Nets,  Markov  Random  Fields  or  Factor  Graphs 


•  Belief  in  a  state  Xj  of  a  node  X  is  modeled  as 

<f>w 

<t>Y 

mw,xr ' 

\  mY,X 

X 


WeN(X) 


set  of  immediate  neighbors  of  X 


ft 


x 


Local 

Evidence 


In  terms  of  “local”  evidence  tpx(x)  and  “messages”  mwxUom  other  nodes 

mW,X  (xj  )  =  ^  &W  (wi  W,X  ’  Xj  )  mY,W  (wi ) 

YsN(i)\j 

=  \r(W,X))f(r(W,X))  n mY,wix) 

wt  r  YgN  (i)\j 

•  Joint  belief  is  given  as 

bw,x  (Xi  ,Xj)  =  k  y/w,x  (wi > Xj  )<f>w  (w,  )(/>x  (Xj )  FK^)  Ylmz  X  (xj  ) 

YeN(W)\X  ZeN(X)\W 
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Expanded  from  J.S.  Yedida,  W.  T.  Freeman  &  Y.  Weiss,  “Understanding  belief 
propagation  and  ts  generalization  ”  (2002) 


Characteristic  Process  Flow 
Across  the  Data  Fusion  “Levels” 


•  Collection  Management 

•  Data  Mining 

•  Fusion  Process  Control 

•  Threat  Assessment 
Course  of  Action  Analysis 
Outcome  Prediction 


Force  Structure  Analysis 
Situation  Analysis 
Scene  Understanding 

Target  Recognition 
Target  Tracking 


•  Feature  Extraction 

•  Signal  Definition 


T  Plans 

1  Goals 

Level  4 

Sys  Characterization/Process  Refinement 


Impact 

Level  3 
Impact  (Threat)  Assessmen 


Plans,  Sub-Goals 


t 


Situations 


Utility 


Level  2 

Situation  Assessment 


Objects 

Situations 

Level  1  J 

r 

Object  Assessment 


”1 

r  Signals/Features 

Objects 

/  1 

Level  0  i 

\ 

Signal/Feature  Assessment 


Signals/Fea  tures 
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Data  Fusion  Functions 
in  Threat  Assessment 


Pre- 

Attack 


Imminent 

Attack 


Attack 

Under¬ 

way 


Post- 

Attack 


Threat  prediction:  Determining  likely  threats:  who,  what, 
where,  when,  why,  how 


Indications  &  warning:  Recognition  that  attack  is 
imminent  or  underway 

Threat  entity  detection  &  characterization:  identification, 
attributes,  composition,  location/track,  activity  capability, 
intent 

Attack  assessment: 

-  Responsible  country/  organization 

-  Intended  target(s) 

-  Intended  effect  (e.g.  damage,  mass  murder,  weaken 
will,  economic  disruption) 

-  Threat  capability  (e.g.  weapon  characterization) 

-  Raid  size,  coordination  &  tactics 

Consequence  assessment 


17 


C4b  Impact  Assessment  (JDL  Level  3) 


Estimation  &  prediction  of 
effects  on  situations  of 
planned  or  estimated/ 
predicted  actions  by  the 
participants 


Including  interactions  between 
action  plans  of  multiple  players: 

>  Assessing  susceptibilities  & 
vulnerabilities  to  estimated/  predicted 
threat  actions  given  one’s  own 
planned  actions 

>  Effect  on  own  forces  and  assets  of 
Intended  course  of  action 


Estimate/predict  threat  knowledge, 


perception  and  planning 


Estimate/predict  effects  of  our  planned 
actions  on  threat  behavior 
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Example:  Air  Strike  Scenario 


STRATEGIC  ATTACK 

PREPARATION  PREPARATION  ATTACK 


POST 

ATTACK 
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Operational  Net  Assessment  (ONA): 

Reasoning  from  Multiple  Perspectives 
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Elements  of  State  Estimation 


Table  A 


Object 

Aspect 

Attributive 

State 

Relational 

State 

Discrete 

Continuous 

Discrete 

Continuous 

Physical 

•  Type,  ID 

•  Activity  State 

•  Location/ 
Kinematics 
•Waveform 
Parameters 

•Causal 

Relation  Type 
•  Role 

Allocation 

» Spatio / 
Temporal 
Relationships 

Informational 

•  Available  Data 
Types 

•  Available  Data 
Records  and 
Quantities 

•Available  Data 
Values 
•Accuracies 
•  Uncertainties 

•  Informational 
Relation  Type 

•  Info  Source/ 
Recipient  Role 
Allocation 

» Source  Data 

-  Quality 

-  Quantity 

-  Timeliness 
» Output  QQT 

Perceptual 

•  Perceptions* 

•  Goals 

•  Priorities 

•  Perceptions* 

•  Cost 

Assignments 

•  Confidence 

•  Plans/Schedule 

•  Perceptions* 

•  Influence 
Relation  Type 

•  Influence 
Source/ 
Recipient  Role 

» Perceptions* 

» Source  ^ 

*  Perceptions  structured  per  Table  A 
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Agent  /  State 


Physical 

Physical  Resources 
Security,  etc. 


Generic  Intentional 
Threat  Model 

Objective  World  Situation 


Informational 

Data  bases 
Data  sources 

Perceptual 

•  Beliefs  concerning  world 
current  &  future  physical  state 

•  Beliefs  concerning  reliability  of 
available  information 

•  Beliefs  concerning  physical, 
informational  &  perceptual 
states  of  other  agents 

•  Goals,  constraints  &  threats 

•  Plans  &  predicted  outcomes 


Capability 


Opportunity 


Intent 


Agent  j  State 


Physical 

Physical  Resources 
Security,  etc. 


Informational 

Data  bases 
Data  sources 


Perceptual 

•  Beliefs  concerning  world 
current  &  future  physical  state 

•  Beliefs  concerning  reliability  of 
available  information 

•  Beliefs  concerning  physical, 
informational  &  perceptual 
states  of  other  agents 

•  Goals,  constraints  &  threats 

•  Plans  &  predicted  outcomes 


~~ZZ~ 


Expanded  from  G.  Rogova  and  E.  Little,  “Ontology  for  Threat  Assessment”  (2004) 


Generic  Intentional 
Threat  Model 

Objective  World  Situation 


Agent  /  State 


Physical 

Physical  Resources 
Security,  etc. 

Informational 

Data  bases 
Data  sources 


Perceptual 

•  Beliefs  concerning  wcdd 
current  &  future  physical^taf 

•  Beliefs  concerning  reliability  ot 
available  information 

•  Beliefs  concerning  physical, 
informational  &  perceptual 
states  of  other  agents 

•  Goals,  constraints  &  threats 

•  Plans  &  predicted  outcomes 

“Z3“ 


Capability 


Opportunity 


Intent 


Interaction 

(e.g.  /  attack  j) 


Direction 


Execution 


Outcomes 


P/rceptual 

fliefs  concerning  world 
currant  &  future  physical  state 
Beliefs  concerning  reliability  of 
Available  information 
Beliefs  concerning  physical, 
informational  &  perceptual 
states  of  other  agents 
•  Goals,  constraints  &  threats 
Plans  &  predicted  outcomes 


Model  of  Potential  Threat 

(Potential  for  Intentional  Actions) 


C%RC 


Model-Based  Evidential  Accrual 
for  Situation/Threat  Assessment 


Google 


Estimated  information  acquisition 
situation 


Live 

Unknown 

Situation 


ATTRIBUTE/ 

ATTRIBUTE/ 

ATTRIBUTE/ 

RELATION 

RELATION 

RELATION 

EXTRACTION 

MATCHING 

GENERATION 

Hyp  Generation 
Hyp  Evaluation 
Hyp  Selection 


Interpretation 


Hypotheses 


Selected  Situation 

Interpretation 

Hypothesis 


Situation 


Oppor¬ 

tunities 


Defende 

Assets 


apon 
(etc. 
Resources 


Sensor/ 
Source, 
Target  & 
Situation 
Models 


MODEL 

MANAGEMENT 


Abduction 

Induction 
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Situation  Assessment 
Technical  Issues  (1  of  2) 


•  Data  Alignment 

-  Exploiting  heterogeneous  information  types  (text,  signals,  numeric 
data,  etc.) 

-  Normalizing  confidence:  source  characterization,  data  evaluation 

•  Hypothesis  Generation 

-  Determining  relevant  data  (data  batching) 

-  Generating/selecting  candidate  scenario  hypotheses  (abduction) 

-  Representing 

»  Relationships  (discrete  &  continuous) 

»  Situations 
»  “Threats”/”lmpacts” 

»  Situation  dynamics 

»  Capabilities,  opportunities,  intents,  perceptions  (etc.) 

•  Hypothesis  Evaluation 

-  Managing  uncertainties  in  data  association 
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Situation  Assessment 
Technical  Issues  (2  of  2) 


•  Hypothesis  Selection 

-  Efficient  schemes  for  representing  and  searching  hypotheses 
(Managing  hypothesis  topologies) 

-  Pruning  schemes 

•  State  Estimation/Prediction 

-  Representation  (Ontology) 

-  Combining  heterogeneous  data  and  uncertainties 

-  Belief  propagation 

»  In  unconstrained  graphs 
»  With  continuous  attribute  &  relational  variables 


How  achieve  formal  rigor  to  ensure  consistency  & 

predictable  performance 

Without  high-confidence  prior  models? 
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Ontology  Requirements 


•  Criteria  for  recognizing  relationships 

-  Including  logical,  semantic,  causal  and  conventional  (e.g. 
moral,  legal,  cultural)  expectations 

-  Requires  a  validated,  comprehensive  ontology 

-  Must  capture  uncertainties  in  dependencies  (e.g.  in  terms  of 
fuzzy  membership) 

•  Criteria  for  recognizing  situations 

-  Some  formal  method  of  situation  semantics  and  situation  logic 

•  Criteria  for  contextually  conditioned  estimation  of  target 
states  and  relationships 

-  Requires  some  form  of  inferential  calculus,  in  which 
uncertainties 

»  In  the  ontology 
»  in  sensor  reports 
»  in  the  inference  process 

are  systematically  represented  and  manipulated 
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Situation  Assessment 
Engineering  Issues 


Architecture  Partitioning 

-  By  domain? 

-  By  resolution  level? 

-  Distributed  processing? 


Human/machine  task  allocation? 


>  Human  vs.  automated  tasks  & 
interfaces 

>  Information  presentation 

Model  Management:  Situations, 
Threats,  Sources,  System 
behavior  &  performance 

-  Ontology  management 

-  Empirical  model  refinement 
abduction,  induction 


C4b  Problem  of  Source  Characterization 


Cases  of  Biased  Information  or 
Biased  Response 

Examples 

Trust¬ 

worthiness 

Fully  cooperative  autonomous 
agents  (unintentional  reporting  & 
response  errors) 

Integrated,  fully  controlled 
sy  ste  m  -of -sy  ste  m  s 

Semi-cooperative  agents  (possibly 
intentional  reporting  &  response 
errors) 

Employees,  Partners 

Deceptive  hostile  agents  (likely 
biased  reporting  &  response) 

Enemy  decoys,  False- 
target  jamming,  Info  ops 

Non-deceptive  hostile  information 
sources  (only  unintentional  reporting 
&  response  errors) 

Enemy  radars,  Combatant 
platforms 

Non-cooperative  third-parties 
sources  (possibly  biased  reporting  & 
responses) 

Commercial  news 
sources,  Websites 

The  fusion  system  must  characterize  the  reliability  &  performance 
of  EACH  source  using  the  ensemble  of  information 
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Open  Net  Concept 


*  COOPERATION/COLLABORATION  (e.q.  in  NCW) 

•  Information  Exchange 


•  Physical  Collaboration,  Cooperation 


•  COMPETITION/CONFLICT 

•  Information  Interception  or 
Disruption  (e.g.  Deceptive 
Information,  Jamming) 

“AGENT  B”  *  Physical  Disruption,  Damage, 

Destruction 


“AGENT  A” 


Can’t  assume  clear  system  boundaries:  “ us ”  vs  “them”: 
Trustworthiness  is  a  matter  of  Estimation 


C%RC 


Summary 


•  Use  the  structure  of  situations  for  inferring  prior  and 
posterior  statistics,  both  for  "attributive"  and  "relational" 
states 

-  Second-order  non-monotonic  predicate  calculus  (e.g.  probabilistic 
infon  notation) 

-  Supports  Operational  Net  Assessment:  Multi-Perspective, 
Conditional,  Counterfactual  Reasoning 

•  Noisy,  complex  and  poorly-modeled  problems: 

-  Requires  integration  of  Abductive,  Inductive  &  Deductive  methods 

-  Requires  Ontology  of  Relationships  &  Situations,  recognizing 
Fuzziness:  HUMAN  BEHAVIOR  IS  COMPLEX ,  BUT  ITS  NOT  RANDOM 

•  Need  sophisticated  techniques  to  characterize  info  sources 

-  Trustworthiness  is  a  matter  of  Estimation 
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Comparison  of  Level  1  and  Level  2/3 
Data  Fusion  Problems 


Level  1  Fusion 
(Object  Assessment; 
e.g.  ATR,  Target  Tracking) 

Level  2/3  Fusion 
(Situation/  Impact  Assessment) 

Level  2/3 
Problems  are 
more 

Strong  constraints  on  relevant 

Weak  constraints  on  relevant 

evidence 

evidence 

-»  Diverse 

Hypothesis  Generation  via 

Hypothesis  Generation  via 

Validation  Gate 

Situation/Behavior  Model 

Strongly  Constrained  Ontology 

Weak  Ontological  Constraints 

-» Complex 

Strong  (i.e.  high-confidence)  Causal 

Weak  Causal  Models:  Typically 

Models:  Typically  dominated  by 

dominated  by 

Physical  Models: 

Human/Group  Behavior  Models: 

e.g.  Signature,  Kinematics 

e.g.  Coordination/  Collaboration, 

->  Uncertain 

models 

Perception,  Value,  Influence  models 
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